Method and apparatus for user interface using gaze interaction

ABSTRACT

A method and apparatus for a user interface using a gaze interaction is disclosed. The method for the user interface using the gaze interaction may include obtaining an image including eyes of a user, estimating a gaze position of the user, using the image including the eyes of the user, and determining whether to activate a gaze adjustment function for controlling a device by a gaze of the user, based on the gaze position of the user with respect to at least one toggle area on a display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority benefit of Korean PatentApplication No. 10-2013-0018394, filed on Feb. 21, 2013, in the KoreanIntellectual Property Office, the disclosure of which is incorporatedherein by reference.

BACKGROUND

1. Field

Example embodiments relate to a method and apparatus for a userinterface using a gaze interaction.

2. Description of the Related Art

Currently, a significant number of gaze interaction technologies, usingat least two external light sources and at least two cameras, may beused to estimate a gaze of a user.

With the development of information technology, gaze interactiontechnologies are being utilized in a variety of fields. For example, inthe advertisement field, advertising draft plans are shown to consumersto determine whether consumers place a greater emphasis on a product, acommercial model, or a company brand by collecting information about aposition at which the consumers gaze and information about a period oftime consumers gaze at the advertising draft plans. Also, in themarketing field, multiple contents are provided to a device, and a userpreference grade may be marked highly with respect to contents peoplegaze at for a long period of time with interest. Such gaze interactiontechnology is expected to be applied in an even greater variety offields. Accordingly, extensive research is being conducted to recognizea gaze of a user more accurately and to use the interaction recognitiontechnology more conveniently.

SUMMARY

The foregoing and/or other aspects are achieved by providing a methodfor a user interface using a gaze interaction, the method includingobtaining an image including eyes of a user, using a camera, estimatinga gaze position of the user, using the image including the eyes of theuser, and determining whether to activate a gaze adjustment function forcontrolling a device by a gaze of the user, based on the gaze positionof the user with respect to at least one toggle area on a display.

The obtaining of the image including the eyes of the user may includeobtaining an image including the eyes of the user, based on at least twointernal light sources included in the display.

The at least one toggle area may include at least one of a plurality ofobjects displayed on the display.

The estimating of the gaze position of the user may include extractingrespective positions of at least two glints, an area in which the atleast two internal light sources are displayed on the eyes of the user,respectively, based on the image including the eyes of the user,calculating a position of a cornea center of the eyes of the user, usingrespective positions of the at least two internal light sources and therespective positions of the at least two glints, calculating respectivepositions of a pupil center of the eyes of the user, using therespective positions of the cornea center of the eyes of the user,calculating respective directions of an optical axis of the eyes of theuser, using the respective positions of the cornea center and therespective positions of the pupil center, calculating respectivedirections of a visual axis of the eyes of the user, based on therespective directions of the optical axis, and identifying the gazeposition, using an intersection point of respective visual axes of theeyes of the user.

The method for the user interface using the gaze interaction may furtherinclude removing noise by at least one external light source.

The removing of the noise by the at least one external light source mayinclude analyzing an image including the eyes of the user, andseparating the at least two internal light sources and the at least oneexternal light source, using a result of the analysis.

The removing of the noise by the at least one external light source mayfurther include changing at least one of a position, a color, and ashape of the at least two internal light sources, and estimating thegaze position of the user, based on the at least two internal lightsources changed.

The determining of whether to activate the gaze adjustment function mayinclude determining whether the gaze position matches the at least onetoggle area, and setting the gaze adjustment function to be anactivation mode or a deactivation mode, based on a result of thedetermination.

The determining of whether the gaze position matches the at least onetoggle area may further include determining whether the gaze positionmatches the at least one toggle area during a predetermined period oftime.

The determining of whether the gaze position matches the at least onetoggle area may further include determining whether the gaze positionmatches the at least one toggle area, based on a predetermined pattern.

The determining of whether to activate the gaze adjustment function mayfurther include setting a default value of the gaze adjustment functionto be the deactivation mode, setting the gaze adjustment function to bethe activation mode during the predetermined period of time when thegaze position matches the at least one toggle area, based on a result ofthe determination.

The method for the user interface using the gaze interaction may furtherinclude setting at least two vertex areas from among a plurality ofvertex areas of the display to be the at least two internal lightsources.

The foregoing and/or other aspects are achieved by providing a methodfor a user interface using a gaze interaction, the method includingobtaining an image including eyes of a user, using at least two internallight sources included in a display and a camera, and estimating a gazeposition of the user, using the image of the eyes of the user.

The foregoing and/or other aspects are achieved by providing a systemfor a user interface using a gaze interaction, the system including adisplay, a camera, an image obtaining unit to obtain an image includingeyes of a user, using the camera, a gaze position estimating unit toestimate the gaze position of the user, using the image including theeyes of the user, and a mode determining unit to determine whether toactivate a gaze adjustment function for controlling a device by the gazeof the user, based on the gaze position of the user with respect to atleast one toggle area on the display.

The foregoing and/or other aspects are achieved by providing a systemfor a user interface using a gaze interaction, the system including adisplay, at least two internal light sources included in the display, acamera, an image obtaining unit to obtain an image including eyes of auser, using the at least two internal light sources and the camera, anda gaze position estimating unit to estimate the gaze position of theuser, using the image including the eyes of the user.

The foregoing and/or other aspects are achieved by providing a userinterface method in a mobile device having a display. The methodincludes displaying an icon on the display while a user waits forcontent to be loaded onto the display, and estimating a gaze position ofthe user by using the icon as a calibration point.

The foregoing and/or other aspects are achieved by providing a method ofunlocking content of a mobile device having a display with a pluralityof toggle areas. The method includes monitoring a gaze position of auser, determining, based on the monitored gaze position, whether theuser has gazed at the toggle areas according to a predetermined pattern,and unlocking the content of the mobile device when it has beendetermined that the user has gazed at the toggle areas according to thepredetermined pattern.

The foregoing and/or other aspects are achieved by providing a method oflocking content of a mobile device having a display with a plurality oftoggle areas. The method includes monitoring a gaze position of a user,determining, based on the monitored gaze position, whether the user hasgazed at the toggle areas according to a predetermined pattern; andlocking the content of the mobile device when it has been determinedthat the user has gazed at the toggle areas according to thepredetermined pattern.

The foregoing and/or other aspects are achieved by providing a method ofchanging content of a mobile device having a display with a plurality oftoggle areas. The method includes monitoring a gaze position of a user,determining, based on the monitored gaze position, whether the user hasgazed at the toggle areas according to a predetermined pattern,activating an activation mode of the mobile device when it has beendetermined that the user has gazed at the toggle areas according to thepredetermined pattern; and changing the content of the mobile devicebased on the monitored gaze position of the user and a predeterminedrule when the activation mode has been activated.

Additional aspects of embodiments will be set forth in part in thedescription which follows and, in part, will be apparent from thedescription, or may be learned by practice of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects will become apparent and more readilyappreciated from the following description of embodiments, taken inconjunction with the accompanying drawings of which:

FIG. 1 illustrates a method for a user interface using a gazeinteraction according to example embodiments;

FIG. 2 illustrates estimating of a gaze position according to exampleembodiments;

FIGS. 3 and 4 illustrate a method for a user interface using a gazeinteraction according to example embodiments; and

FIG. 5 illustrates an apparatus for a user interface using a gazeinteraction according to example embodiments.

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings, wherein like referencenumerals refer to the like elements throughout. Embodiments aredescribed below to explain the present disclosure by referring to thefigures.

FIG. 1 illustrates a method for a user interface using a gazeinteraction according to example embodiments.

Referring to FIG. 1, in operation 110, the method for the user interfaceusing the gaze interaction may obtain an image, including eyes of auser, using a camera. Here, the camera may be included in a device, ormay be disposed outside the device. The device may be configured as adisplay or in a plurality of other configurations, and may include amobile device, a television (TV), a tablet, a personal computer (PC)monitor, a digital information display, a medical image display, aholographic display, a multi-view three-dimensional (3D) display, andthe like. Also, the term “display” may refer to an apparatus forrepresenting data such as a character, a diagram, or an image in thedevice.

The camera may capture an image including the eyes of the user. In thisinstance, a quantity of cameras capturing the image including the eyesof the user may be at least one camera.

The method for the user interface using the gaze interaction may obtainthe image including the eyes of the user, based on a light source torecognize a gaze position of the user precisely. More particularly, themethod for the user interface using the gaze interaction may obtain theimage including the eyes of the user using the camera, based on at leasttwo internal light sources. Here, the at least two internal lightsources may include a point light source in the display. Moreparticularly, the method for the user interface using the gazeinteraction may set at least two vertex areas, from among a plurality ofvertex areas of the display, to be at least two internal light sources.For example, when the display is a square, the method for the userinterface using the gaze interaction may set two vertex areas from amongfour vertex areas to be internal light sources. Accordingly, powerefficiency may be enhanced in a system requiring low power such as amobile device, and the like, using light emitted from the display as alight source, without using an additional light source. Also, noisetypically generated by an external light source may be removed moreeasily using an internal light source. For example, in a case of themobile device, a direction of a camera of the mobile device may faceupward because a user generally faces a display portion of the mobiledevice in relation to an upward direction of the display portion. Hence,there may be an increased probability of a camera of being exposed to anexternal light source such as an internal lightning, and the like, andthereby an increased occurrence of noise. In this instance, the methodfor the user interface using the gaze interaction may determineinformation about an image shown on the display in advance.Transitively, separating the internal light sources and the externallight sources in the display may be performed more efficiently.

In operation 120, the method for the user interface for the gazeinteraction may estimate a gaze position of a user, using an imageincluding eyes of the user. More particularly, the method for the userinterface using the gaze interaction may extract a position of at leasttwo glints, based on the image including the eyes of the user. Here, theat least two glints may refer to one or more areas in which reflectionsof the at least two internal light sources are displayed on the eyes ofthe user, respectively. Accordingly, a quantity of glints for a singleeye may correspond to a quantity of internal light sources. For example,when the number of internal light sources used is two, the internallight sources may be displayed on the eyes, respectively. Therefore, thenumber of glints for the single eye may be equal to the number ofinternal light sources, in this example, two.

Also, the method for the user interface for the gaze interaction maycalculate a position of a cornea center of the eyes, respectively, usingthe position of the at least two internal light sources and the positionof the at least two glints. More particularly, the method for the userinterface using the gaze interaction may extract a first plane includinga first internal light source, from one of two internal light sources, acornea surface on which the first internal light source is displayed,and a first glint image corresponding to the first internal lightsource. Further, a second plane corresponding to a second internal lightsource, from another of the two internal light sources, may beextracted, and the cornea center may be calculated using an intersectionpoint of the first plane and the second plane.

More particularly, the first plane may be extracted based on Equation 1because a cornea may perform the same role as a reflection surface withrespect to a light source, an incident ray and a reflected ray may be onthe same plane based on a law of reflection, and an incidence angle anda reflection angle may be equal.

r ₁=2(n ₁ ·l ₁)n ₁ −l ₁  [Equation 1]

Here, r₁ denotes a unit vector in a G_(1img) direction representing aposition of a first glint image. Here, G_(1img) represents a cornealreflex generated in a camera by a first internal light source. l₁denotes a unit vector in a (L₁−G₁) direction wherein L₁ represents thefirst internal light source of the two internal light sources, and G₁represents a normal of the cornea surface at an index point with respectto the first internal light source. n₁ denotes a normal vector at anindex point in (G₁−C) direction wherein C represents a cornea center.The method for the user interface using the gaze interaction maygenerate the first plane including the first internal light source, thecornea surface on which the first internal light source is displayed,and the first glint image, using r₁.

The second plane may be extracted using Equation 2 as the first plane isextracted.

r ₂=2(n ₂ ·l ₂)n ₂ −l ₂  [Equation 2]

Here, r₂ denotes a unit vector in a G_(2img) direction representing aposition of a second glint image. Here, G_(2img) represents a cornealreflex generated in a camera by a second internal light source. l₂denotes a unit vector in (L₂−G₂) direction wherein L₂ represents thesecond light source, and G₂ represents a normal of a cornea surface atan index point with respect to the second light source. n₂ denotes anormal vector at an index point of a (G₂−C) direction. The method forthe user interface using the gaze interaction may generate the secondplane, using r₂.

The method for the user interface using the gaze interaction mayestimate an interaction line R_(cornea) of the first plane and thesecond plane, using the first plane and the second plane. Also, themethod for the user interface using the gaze interaction may calculate acorneal curvature radius r_(c), to extract the cornea center. Here,r_(c) may be calculated based on Equation 3.

|G ₂ −C _(L2) |=|G ₁ −C _(L1) |=r _(c)  [Equation 3]

Here, C_(L1) and C_(L2) denote a cornea center estimated using the firstlight source and the second light source. C_(L1), C_(L2), and C may beequal to one another as defined in C_(L1)=C_(L2)=C. Accordingly, themethod for the user interface using the gaze interaction may generater_(c) based on Equation 3 and that C_(L1)=C_(L2)=C. Also, the method forthe user interface using the gaze interaction may calculate a positionof a cornea center C, using R_(cornea) and r_(c).

The method for the user interface using the gaze interaction maycalculate a position of a pupil center of the eyes, respectively, usingthe position of the cornea center of the eyes, respectively. Moreparticularly, when a pupil is assumed to be included in a predeterminedplane Π, the pupil center may be disposed on the plane Π. The plane Πmay be represented based on Equation 4 because a distance between thecornea center C and a pupil center E may be obtained by |E−C|.

$\begin{matrix}{{{\frac{\left( {E - C} \right)}{{E - C}} \cdot \left\lbrack {x - C} \right\rbrack} + {{E - C}}} = 0} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack\end{matrix}$

Here, x denotes a three-dimensional (3D) point x=(x, y, z) with respectto a camera. The method for the user interface using the gazeinteraction may calculate lights reflected off of a cornea {f_(k)|k=1 .. . n}, and determine {P_(k)|k=1, . . . , n} representing contourspoints of the pupil, using {f_(k)|k=1 . . . n} because the pupil mayproject a light reflected off of the cornea. A distance between therespective contour points of the pupil and the pupil center may beconsistent, and the distance being consistent may be represented asEquation 5.

|P ₁ −E|=|P ₂ −E|= . . . =|P _(n) −E|  [Equation 5]

Here, |P_(k)−E| denotes a distance between contour points of a pupilP_(k) and the cornea center E. Accordingly, the method for the userinterface using the gaze interaction may calculate the cornea center Esatisfying Equations 4 and 5.

The method for the user interface using the gaze interaction maycalculate a direction of an optical axis of the eyes, respectively,using a positon of the cornea center and a position of the pupil center.Here, the term “optical axis” may refer to a direction of anatomicaleyes. Also, optical axis may refer to a line linking the pupil centerand the cornea center, and therefore, the method for the user interfaceusing the gaze interaction may calculate the direction of the opticalaxis by extracting the line linking the pupil center and the corneacenter.

The method for the user interface using the gaze interaction maycalculate a direction of a visual axis of the eyes, respectively, basedon the direction of the optical axis. Here, the visual axis may refer toa direction in which a user actually gazes at an object. The visual axismay refer to a straight line linking a fovea and the cornea centerbecause an image may be displayed on the fovea by refraction of thecornea.

More particularly, an angle difference between the optical axis and thevisual axis may exist by an amount of an angular offset β. The angularoffset β may be calculated based on Equation 6.

$\begin{matrix}{{\cos \; \beta} = \frac{\left( {E - C} \right) \cdot \left( {P_{c} - C} \right)}{{E - {C\left. {P_{c} - C} \right)}}}} & \left\lbrack {{Equation}\mspace{14mu} 6} \right\rbrack\end{matrix}$

Here, P_(c) denotes a calibration point. More particularly, a shape ofeyes and a refractive index of a cornea may differ from users.Accordingly, the method for the user interface using the gazeinteraction may define a line linking the calibration point P_(c) and acornea center C to be a visual axis, and using the defining of the line,the angular offset β may be calculated. The visual axis differing by theamount of the angular offset β may be calculated using the calculatedangular offset β.

The method for the user interface using gaze interaction may generatethe visual axis of the eyes, respectively, and identify a gaze positionof the user, using an interaction point of the visual axis of the eyes,respectively. For example, the method for the user interface using thegaze interaction may calculate a direction of a first visual axis, beinga visual axis of a left eye of the user and a direction of a secondaxis, being a visual axis of a right eye of the user. An intersectionpoint of the first visual axis and the second visual axis may beextracted, and a position of the intersection point may be set to be anactual gaze position of the user.

The method for the user interface using the gaze interaction may removenoise by at least one external light source. More particularly, themethod for the user interface using the gaze interaction may analyze animage including eyes of a user. At least two internal light sources andthe at least one external light source may be separated, using a resultof the analysis. For example, when a glint is generated by an externallight source, the method for the user interface using the gazeinteraction may determine a configuration, a color, a shape, and thelike, of an image by analyzing the image including the eyes of the user.The glint by the external light source may be identified by separatinginternal light sources from the external light source. The glint may beremoved by the external light source, or the noise by the external lightsource may be removed by performing a gaze interaction using only glintsby internal light sources rather than the glint by the external lightsource.

When noise occurs by the external light source, the method for the userinterface using the gaze interaction may change at least one of aposition, a color, and a shape of the at least two internal lightsources, and estimate a gaze position of the user, based on the at leasttwo internal light sources changed. For example, when distinguishing theglint by the external light source from the glints by the internal lightsources is difficult, the method for the user interface using the gazeinteraction may change at least one of the position, the color, and theshape of the at least two internal light sources so as to bedistinguished from the external light source. A precise gaze interactionfrom which noise has been removed using only the glints by the at leasttwo changed internal light sources may be performed.

In operation 130, the method for the user interface using the gazeinteraction may determine whether to activate a gaze adjustmentfunction, based on a gaze position of a user with respect to at leastone toggle area on a display. Here, the gaze adjustment function mayrefer to a function for controlling an interface of a device, using agaze. The at least one toggle area may refer to a predetermined areapresent on the display for determining whether to activate or deactivatethe gaze adjustment function. Accordingly, the user may determinewhether to activate the gaze adjustment function with a gaze only,rather than using a hand of the user or another device such as a remotecontrol or a stylus.

The at least one toggle area may include at least one calibration point.The gaze position of the user may be estimated more precisely, using theat least one toggle area as the at least one calibration point.

Additionally, the at least one toggle area may include at least one ofthe plurality of objects displayed on the display. Here, the pluralityof objects may refer to a total of data, such as a character, a diagram,an image, and the like, displayed on the display. For example, the atleast one toggle area may include a status bar displayed on the display.Here, the status bar may refer to a configuration providing informationon a status of a device to the user. Generally, the user may move a gazedirection to the status bar readily because the status bar may bedisplayed on the display at all times. Thus, through more convenient useof a gaze, the gaze adjustment function may be activated or deactivatedby the user.

At least one of the plurality of objects displayed on the display may beused as a calibration point. For example, the at least one calibrationpoint may include a progress bar. Here, the progress bar may refer to aconfiguration representing a status of a progress, for example, a statusof a download. As such, when the progress bar is the calibration point,the user may gaze at a tip of the progress bar, and through calibratingusing the tip of the calibration bar, the method for the user interfaceusing the gaze interaction may estimate the gaze position of the usermore precisely. As another example, the method for the user interfaceusing the gaze interaction may use an icon, for example, an hourglassicon, represented during operation of a mouse or a program as thecalibration point, and may use an icon, for example, a spiral icon,represented during loading of an online page as the calibration point.

The method for the user interface using the gaze interaction maydetermine whether the gaze position matches the at least one togglearea. Here, the matching of the gaze position and the at least onetoggle area may refer to the user gazing at the at least one togglearea. The method for the user interface using the gaze interaction mayset the gaze adjustment function to be in an activation mode or adeactivation mode, based on a result of the determination. For example,the method for the user interface using the gaze interaction mayidentify a current mode of the gaze adjustment function, and based onthe result of the determination whether the gaze position matches the atleast one toggle area, may set a mode of the gaze adjustment function.As a detailed example, when the gaze adjustment function is on thedeactivation mode, and the gaze position matches the at least one togglearea, the mode of the gaze adjustment function may be changed to theactivation mode. In this instance, when the gaze position fails to matchthe at least one toggle area, the deactivation mode may be maintained.When the user gazes at the at least one toggle area on the activationmode, the mode of the gaze adjustment function may be changed to thedeactivation mode. As another example, when the gaze adjustment functionis on the deactivation mode, the mode of the gaze adjustment functionmay be changed to the activation mode through the user gazing at the atleast one toggle area. The mode of the gaze adjustment function may bechanged to the deactivation mode, when the user fails to gaze at the atleast one toggle area during a predetermined activation mode, forexample, for a time of one second.

The method for the user interface using the gaze interaction maydetermine whether the gaze position matches the at least one toggle areaduring a predetermined period of time. For example, when thepredetermined time is three seconds, the method for the user interfaceusing the gaze interaction may determine that the gaze position fails tomatch the at least one toggle area when the user gazes at the at leastone toggle area for two seconds.

The method for the user interface using the gaze interaction maydetermine whether the gaze position matches the at least one togglearea, based on a predetermined pattern. For example, when a number oftoggle areas is three, for example, a first toggle area, a second togglearea, and a third toggle area, the predetermined pattern matches thegaze position in a sequential manner of “the third toggle area-the firsttoggle area-the second toggle area”, the method for the user interfaceusing the gaze interaction may determine that the gaze position matchesthe toggle areas only when the user gazes at the toggle areas, based onthe predetermined pattern. Accordingly, when the user gazes at thetoggle areas in the sequential manner of “the third toggle area-thesecond toggle area-the first toggle area”, the gaze position may bedetermined not to match the toggle areas.

The method for the user interface using the gaze interaction may set adefault value of the gaze adjustment function to be the deactivationmode. The setting of the default value may refer to maintaining thedeactivation mode when the gaze position fails to match the at least onetoggle area. The method may determine whether the gaze position matchesthe at least one toggle area, and as a result of the determination, whenthe gaze position matches the at least one toggle area, the gazeadjustment function may be set as the activation mode for apredetermined period of time. For example, as a default, the gazeadjustment function may be set as the deactivation mode. Here, when theuser gazes at the at least one toggle area, the mode of the gazeadjustment function may be changed to the activation mode for apredetermined period of time, for example, two seconds. When thepredetermined period of time has been completed, the mode of the gazeadjustment function may be changed to the deactivation mode, such as, adefault value.

FIG. 2 illustrates estimating of a gaze position according to exampleembodiments.

Referring to FIG. 2, in 210, a mobile device may include two internallight sources 211 and 212, a toggle area 221, and a calibration point222. In addition, a camera 217 may be included outside of the display.More particularly, the two internal light sources 211 and 212 may bedisplayed on eyes of a user, and two glints 213 and 214, correspondingto the two internal light sources 211 and 212, may occur on a left eyeof the user, and two glints 215 and 216, also corresponding to the twointernal light sources 211 and 212, may occur on a right eye of theuser. The two internal light sources 211 and 212 may each include, as anexample, an LED or pixel emitting light at a particular wavelength. Themethod for the user interface using the gaze interaction may obtain animage including the eyes of the user on which the glints occur, usingthe camera. A gaze position of the user may be identified by calculatinga position of the glints, a position of a cornea center, a position of apupil center, a direction of an optical axis, and a direction of avisual axis, based on the image obtained. Here, the calibration point222 may be disposed on an upper edge of the toggle area 221. As anexample, the method for the user interface using the gaze interactionmay allow the user to gaze at the calibration point 222 by blinking atthe calibration point 222, and through this, a more precise gazeposition may be estimated.

In 250, in the mobile device, a camera 257 may be included outside aperimeter of the display as shown in 210, and two internal light sources251 and 252 may be included within the perimeter of the display.Furthermore, the mobile device in 250 may include a progress bar 261.Here, the progress bar 261 may be used as a toggle area. A calibrationpoint 262 may be disposed on a tip of the progress bar 261. For example,when the progress bar 261 is displayed on the display, the user maynaturally or reflexively gaze at the tip of the progress bar 261.Transitively, the camera 257 may obtain an image of gazing at thecalibration point 262 by setting a portion gazed at by the user to bethe calibration point 262, and the method for the user interface usingthe gaze interaction may precisely estimate the gaze position of theuser.

FIGS. 3 and 4 illustrate a method for a user interface using a gazeinteraction according to example embodiments.

Referring to FIG. 3, the mobile device 310 may include a camera 311, adisplay, and four toggle areas 321 to 324. In various embodiments, thefour toggle areas 321 to 324 may or may not be displayed on the display.

As a detailed example, a user may perform a search for content BB,subsequent to connecting to a hypothetical website AAA, using the mobiledevice 310. The mobile device 310 may display a search result page.Here, when loading of the search result page is delayed, a spiral icon331 may be displayed on the display along with text indicating“loading”. In this instance, the user may naturally or reflexively gazeat the spiral icon 331. Transitively, the method for the user interfaceusing the gaze interaction may use the spiral icon 331 as a calibrationpoint, and thereby more precisely estimate the gaze position of theuser.

A mobile device 350 may display a search result page, subsequent to theloading, and the user may adjust an interface of the mobile device 350,using a gaze adjustment function. More particularly, the user may gazeat toggle areas 371 through 374 in a sequential manner of “the firsttoggle area 371-the second toggle area 372-the fourth toggle area374-the third toggle area 373” for more than a second, respectively,based on a predetermined time, for example, a second, and apredetermined pattern. Transitively, the method for the user interfaceusing the gaze interaction may set the gaze adjustment function to be onan activation mode.

When the gaze adjustment function is on the activation mode, and theuser gazes at a bottom portion of the display, the method for the userinterface using the gaze interaction may slowly scroll down a searchresult page displayed based on a predetermined rule. Alternatively, apredetermined area at which the user gazes may be selected based on thepredetermined rule.

As another example, when the mobile device 350 is in a lock status, themethod for the user interface using the gaze interaction may unlock thelock status of the mobile device 350, using a gaze interaction. Moreparticularly, when the mobile device 350 is in the lock status, the usermay gaze at the four toggle areas, respectively, based on apredetermined period of time and a predetermined pattern. When a gazeposition is determined to match the four toggle areas, the method forthe user interface using the gaze interaction may unlock the lock statusof the mobile device 350. Although four toggle areas have been used inthe preceding example, any quantity or pattern of toggle areas mayalternatively be used.

As still another example, when the mobile device 350 is in an unlockstatus, the method for the user interface using the gaze interaction maylock the unlock status of the mobile device 350, using a gazeinteraction. More particularly, when the mobile device 350 is in theunlock status, the user may gaze at the four toggle areas, respectively,based on a predetermined period of time and a predetermined pattern.When a gaze position is determined to match the four toggle areas, themethod for the user interface using the gaze interaction may lock theunlock status of the mobile device 350. Although four toggle areas havebeen used in the preceding example, any quantity or pattern of toggleareas may alternatively be used. A larger quantity of toggle areas, forexample, may be used to increase security of the lock status.

Referring to FIG. 4, a device 410, for example, a personal computer (PC)monitor or a smart television (TV), may include a camera 411 and adisplay. An online address bar 421, an online page, and two gazeadjustment areas 422 and 423 may be displayed on the display. Here, whenthe gaze adjustment function is set to an activation mode, the gazeadjustment area may refer to an area for adjusting an interface of thedevice 410. In particular, the online address bar 421 may refer to atoggle area and a calibration point. For example, when the user gazes atthe online address bar 421, the gaze position is determined to match atoggle area corresponding to the online address bar 421, through theonline address bar 421 being used as the toggle area, and transitively,the method for the user interface using the gaze interaction mayactivate the gaze adjustment function. The online address bar 421 mayfunction as the calibration point, through the user gazing at a tip ofthe online address bar 421, and transitively, a more precise estimationof the gaze position may be possible. As another example, when an onlinebrowser of the device 410 is in operation, an hourglass icon 431 may bedisplayed on the display. In this instance, the gaze position of theuser may be estimated precisely, through the method for the userinterface using the gaze interaction setting the hourglass icon 431 tobe the calibration point because a gaze of the user may be disposed onthe hourglass icon 431.

On the activation mode of the gaze adjustment function, when the usergazes at a first gaze adjustment area 422, the method for the userinterface using the gaze interaction may capture an image displayed onthe display, based on a predetermined rule. When the user gazes at asecond gaze adjustment area 423, the method for the user interface usingthe gaze interaction may display a subsequent web page of a web pagedisplayed on the display. Also, the user may use a function associatedwith copying, pasting, and the like, of a web page, using the gazeadjustment area. In another embodiment, a user may switch betweenapplications when the user gazes at a first gaze adjustment area 422,the second gaze adjustment area 423, or a predetermined pattern of gazeadjustment areas.

FIG. 5 illustrates an apparatus for a user interface using a gazeinteraction according to example embodiments.

Referring to FIG. 5, the apparatus for the user interface using the gazeinteraction may include a display 510 and a camera 520.

An image obtaining unit 530 may obtain an image including eyes of auser, using the camera 520.

A gaze position estimating unit 540 may estimate a gaze position of theuser, using the image including the eyes of the user.

A mode determining unit 550 may determine whether to activate the gazeadjustment function for controlling the device through a gaze of theuser, based on the gaze position of the user with respect to at leastone toggle area on the display.

For conciseness and ease of description, details pertaining todescriptions discussed with reference to FIGS. 1 through 4 that may beapplied to the apparatus for the user interface using the gazeinteraction according to the example embodiment of FIG. 5 are omitted.

A portable device as used throughout the present specification includesmobile communication devices, such as a personal digital cellular (PDC)phone, a personal communication service (PCS) phone, a personalhandy-phone system (PHS) phone, a Code Division Multiple Access(CDMA)-2000 (1X, 3X) phone, a Wideband CDMA phone, a dual band/dual modephone, a Global System for Mobile Communications (GSM) phone, a mobilebroadband system (MBS) phone, a satellite/terrestrial Digital MultimediaBroadcasting (DMB) phone, a Smart phone, a cellular phone, a personaldigital assistant (PDA), an MP3 player, a portable media player (PMP),an automotive navigation system (for example, a global positioningsystem), and the like. Also, the portable device as used throughout thepresent specification includes a digital camera, a plasma display panel,and the like.

The method for the user interface using the gaze interaction accordingto the above-described embodiments may be recorded in non-transitorycomputer-readable media including program instructions to implementvarious operations embodied by a computer. The media may also include,alone or in combination with the program instructions, data files, datastructures, and the like. Examples of non-transitory computer-readablemedia include magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD ROM discs and DVDs;magneto-optical media such as optical discs; and hardware devices thatare specially configured to store and perform program instructions, suchas read-only memory (ROM), random access memory (RAM), flash memory, andthe like.

Examples of program instructions include both machine code, such asproduced by a compiler, and files containing higher level code that maybe executed by the computer using an interpreter. The described hardwaredevices may be configured to act as one or more software modules inorder to perform the operations of the above-described embodiments, orvice versa. Any one or more of the software modules described herein maybe executed by a controller such as a dedicated processor unique to thatunit or by a processor common to one or more of the modules. Thedescribed methods may be executed on a general purpose computer orprocessor or may be executed on a particular machine such as theapparatuses described herein.

Although embodiments have been shown and described, it would beappreciated by those skilled in the art that changes may be made inthese embodiments without departing from the principles and spirit ofthe disclosure, the scope of which is defined by the claims and theirequivalents.

What is claimed is:
 1. A method for a user interface using a gazeinteraction, the method comprising: obtaining an image including eyes ofa user, using a camera; estimating a gaze position of the user using theimage including the eyes of the user; and determining whether toactivate a gaze adjustment function for controlling a device using agaze of the user, based on a gaze position of the user with respect toat least one toggle area on a display.
 2. The method of claim 1, whereinthe obtaining of the image including the eyes of the user comprises:obtaining an image including the eyes of the user using at least twointernal light sources included in the display.
 3. The method of claim1, wherein the at least one toggle area comprises: at least onecalibration point.
 4. The method of claim 1, wherein the at least onetoggle area comprises: at least one of a plurality of objects displayedon the display.
 5. The method of claim 2, wherein the estimating of thegaze position of the user comprises: extracting respective positions ofat least two glints, being an area in which the at least two internallight sources are displayed on the eyes of the user, respectively, basedon the image including the eyes of the user; calculating a position of acornea center of the eyes of the user, using respective positions of theat least two internal light sources and the respective positions of theat least two glints; calculating respective positions of a pupil centerof each of the eyes of the user, using the respective positions of thecornea center of the eyes of the user; calculating respective directionsof an optical axis of the eyes of the user, using the respectivepositions of the cornea center and the respective positions of the pupilcenter; calculating respective directions of a visual axis of the eyesof the user, based on the respective directions of the optical axis; andidentifying the gaze position, using an intersection point of respectivevisual axes of the eyes of the user.
 6. The method of claim 2, furthercomprising: removing noise by at least one external light source,wherein the removing of the noise by the at least one external lightsource comprises: analyzing an image including the eyes of the user; andseparating the at least two internal light sources and the at least oneexternal light source, using a result of the analysis.
 7. The method ofclaim 6, wherein the removing of the noise by the at least one externallight source further comprises: changing at least one of a position, acolor, and a shape of the at least two internal light sources, andestimating the gaze position of the user, based on the changing of theat least two internal light sources.
 8. The method of claim 1, whereinthe determining of whether to activate the gaze adjustment functioncomprises: determining whether the gaze position matches the at leastone toggle area; and setting the gaze adjustment function to be anactivation mode or a deactivation mode, based on a result of thedetermination.
 9. The method of claim 8, wherein the determining ofwhether the gaze position matches the at least one toggle area furthercomprises: determining whether the gaze position matches the at leastone toggle area within a predetermined period of time.
 10. The method ofclaim 8, wherein the determining of whether the gaze position matchesthe at least one toggle area further comprises: determining whether thegaze position matches the at least one toggle area, based on apredetermined pattern.
 11. The method of claim 8, wherein thedetermining of whether to activate the gaze adjustment function furthercomprises: setting a default value of the gaze adjustment function to bethe deactivation mode; setting the gaze adjustment function to be theactivation mode during the predetermined period of time when the gazeposition matches the at least one toggle area, based on a result of thedetermination.
 12. The method of claim 2, further comprising: setting atleast two vertex areas from among a plurality of vertex areas of thedisplay to be the at least two internal light sources.
 13. A method fora user interface using a gaze interaction, the method comprising:emitting light using at least two internal light sources included in adisplay; obtaining, by way of a camera, an image including eyes of auser, wherein reflections of the at least two internal light sources aredisplayed on the eyes of the user; and estimating a gaze position of theuser, using the image of the eyes of the user.
 14. An apparatus for auser interface using a gaze interaction, the system comprising: adisplay; a camera; an image obtaining unit to obtain an image includingeyes of a user, using the camera; a gaze position estimating unit toestimate the gaze position of the user using the image including theeyes of the user; and a mode determining unit to determine whether toactivate a gaze adjustment function for controlling a device using thegaze of the user, based on a gaze position of the user with respect toat least one toggle area on the display.
 15. An apparatus for a userinterface using a gaze interaction, the system comprising: a display; atleast two internal light sources included in the display; a camera; animage obtaining unit to obtain, by way of the camera, an image includingeyes of a user, wherein reflections of the at least two internal lightsources are displayed on the eyes of the user; and a gaze positionestimating unit to estimate the gaze position of the user, using theimage including the eyes of the user.
 16. A user interface method in amobile device having a display, the method comprising: displaying anicon on the display while a user waits for content to be loaded onto thedisplay; and estimating a gaze position of the user by using the icon asa calibration point.
 17. A method of unlocking content of a mobiledevice having a display with a plurality of toggle areas, the methodcomprising: monitoring a gaze position of a user; determining, based onthe monitored gaze position, whether the user has gazed at the toggleareas according to a predetermined pattern; and unlocking the content ofthe mobile device when it has been determined that the user has gazed atthe toggle areas according to the predetermined pattern.
 18. A method oflocking content of a mobile device having a display with a plurality oftoggle areas, the method comprising: monitoring a gaze position of auser; determining, based on the monitored gaze position, whether theuser has gazed at the toggle areas according to a predetermined pattern;and locking the content of the mobile device when it has been determinedthat the user has gazed at the toggle areas according to thepredetermined pattern.
 19. A method of changing content of a mobiledevice having a display with a plurality of toggle areas, the methodcomprising: monitoring a gaze position of a user; determining, based onthe monitored gaze position, whether the user has gazed at the toggleareas according to a predetermined pattern; activating an activationmode of the mobile device when it has been determined that the user hasgazed at the toggle areas according to the predetermined pattern; andchanging the content of the mobile device based on the monitored gazeposition of the user and a predetermined rule when the activation modehas been activated.
 20. The method of claim 19, wherein thepredetermined rule comprises scrolling content of the mobile devicedownward when the gaze position of the user is monitored as directedtowards a bottom portion of the display of the mobile device.
 21. Themethod of claim 19, wherein the predetermined rule comprises enlargingcontent displayed on a predetermined portion of the display of themobile device when the gaze position of the user is monitored asdirected towards the predetermined portion.